Skip to content

Add compile-time fault-injection infrastructure for stack-walker recovery paths#661

Draft
zhengyu123 wants to merge 10 commits into
mainfrom
zgu/fault-injection
Draft

Add compile-time fault-injection infrastructure for stack-walker recovery paths#661
zhengyu123 wants to merge 10 commits into
mainfrom
zgu/fault-injection

Conversation

@zhengyu123

@zhengyu123 zhengyu123 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?:
Adds a compile-time fault-injection layer that deliberately corrupts pointers at the profiler's real memory-access sites so the existing recovery paths (SafeAccess safefetch and walkVM's setjmp/longjmp) are exercised continuously under test. Entirely gated behind FAULT_INJECTION: when the flag is absent every construct is a strict identity/no-op with zero runtime cost, so production release/debug/asan/tsan builds are unaffected.

To build fault-injected binaries, add -PenableFaultInjection flag to the command line, e.g.

./gradlew -PenableFaultInjection  clean builddebug

Motivation:

Additional Notes:

What's included

  • Core layer (new files)

faultInjection.h — three probability tiers as integer xorshift thresholds (RARE 0.01%, UNLIKELY 0.1%, LIKELY 1%) and the INJECT_FAULT_{ADDRESS,INT,LONG}_{RARE,UNLIKELY,LIKELY} macros. When the flag is off, each macro is a strict identity ((ptr)/(v)).
faultInjection.cpp — PRNG, PROT_NONE guard-region setup, and poisonAddress() (mix of a mmap'd guard-page pointer and random non-canonical addresses).

  • Per-thread PRNG
    threadLocalData.h — per-thread xorshift64 state in ProfiledThread (async-signal-safe), with a global atomic fallback.

  • Injection sites (wrap the dereferenced pointer)
    stackWalker.cpp — walkFP / walkDwarf SafeAccess loads (LIKELY) → safefetch recovery.
    hotspot/hotspotSupport.cpp — walkVM raw derefs (UNLIKELY) → setjmp/longjmp recovery.
    hotspot/vmStructs.h — VMStructs::at() (RARE).
    profiler.cpp — calls faultinj::init() at signal-handler setup.

  • Counters (counters.h, live under -DCOUNTERS)
    safecopy_failed, safefetch_failed, faults_injected, walkvm_longjmp_recovered — instrument the recovery mechanisms so injection activity and each recovery route are observable.

  • Build wiring

-PenableFaultInjection appends -D__FAULT_INJECTION__ to the standard release/debug library builds (so buildDebug/buildRelease produce a fault-injected libjavaProfiler.so) and to the gtest build. Deliberately not applied to asan/tsan/fuzzer, which install their own SIGSEGV interception.

  • Tests
    faultInjection_ut.cpp — disabled-path identity checks (default build) plus enabled-path tier-rate bands and SafeAccess/walkVM recovery tests (-PenableFaultInjection).

How to test the change?:

  • ./gradlew :ddprof-lib:gtestDebug_faultInjection_ut -PenableFaultInjection — 6 enabled cases pass.
  • ./gradlew :ddprof-lib:gtestDebug_faultInjection_ut (default) — 2 identity cases pass.
  • ./gradlew -PenableFaultInjection :ddprof-lib:assembleDebug — verified faultinj::* symbols present (.symtab; hidden visibility, not exported).

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-15401

Unsure? Have a question? Request a review!

@dd-octo-sts

dd-octo-sts Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #29506661506 | Commit: 2fdca40 | Duration: 14m 53s (longest job)

All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - - -
8-ibm - - -
8-j9 - -
8-librca - -
8-orcl - - -
11 - - -
11-j9 - -
11-librca - -
17 - -
17-graal - -
17-j9 - -
17-librca - -
21 - -
21-graal - -
21-librca - -
25 - -
25-graal - -
25-librca - -

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-07-16 14:46:07 UTC

@dd-octo-sts

dd-octo-sts Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit d85f055)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124877517 Commit: d85f055273e38b3bd9ad5a6913516fca7d60a0f2

⚠️ Significant outliers

  • 🔴 future-genetic (JDK 25): runtime +5.5% (2025→2137 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10232 ms (21 iters) ✅ 10166 ms (21 iters) ≈ -0.6% (±11.1%) — / —
akka-uct 25 ✅ 8907 ms (24 iters) ✅ 8929 ms (24 iters) ≈ +0.2% (±9.6%) — / —
finagle-chirper 21 ✅ 5972 ms (33 iters) ✅ 5984 ms (33 iters) ≈ +0.2% (±25.3%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5441 ms (36 iters) ✅ 5530 ms (36 iters) ≈ +1.6% (±25%) ⚠️ W:3 / ⚠️ W:4
fj-kmeans 21 ✅ 2734 ms (69 iters) ✅ 2707 ms (70 iters) ≈ -1% (±2.7%) — / —
fj-kmeans 25 ✅ 2822 ms (66 iters) ✅ 2833 ms (66 iters) ≈ +0.4% (±2.6%) — / —
future-genetic 21 ✅ 2088 ms (89 iters) ✅ 2087 ms (89 iters) ≈ -0% (±2.7%) — / —
future-genetic 25 ✅ 2025 ms (92 iters) ✅ 2137 ms (87 iters) 🔴 +5.5% — / —
naive-bayes 21 ✅ 1286 ms (134 iters) ✅ 1344 ms (128 iters) ≈ +4.5% (±33%) — / —
naive-bayes 25 ✅ 1010 ms (170 iters) ✅ 1027 ms (166 iters) ≈ +1.7% (±32.1%) — / —
reactors 21 ✅ 16543 ms (15 iters) ✅ 16267 ms (15 iters) ≈ -1.7% (±8%) — / —
reactors 25 ✅ 18414 ms (15 iters) ✅ 18374 ms (15 iters) ≈ -0.2% (±4.2%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / 3 2017 / 2006 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 3 2142 / 2423 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 5 / 2 8761 / 8937 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 4 8332 / 8785 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 2 1285 / 1288 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / 3 1273 / 1302 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / 1 3049 / 3118 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 1 2892 / 2926 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 6 3506 / 3515 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 2 3476 / 3495 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 1758 / 1644 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 2 1913 / 1843 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/124877424

@zhengyu123 zhengyu123 changed the title Investigate and implement fault injection for Java profiler Add compile-time fault-injection infrastructure for stack-walker recovery paths Jul 15, 2026
@zhengyu123 zhengyu123 requested a review from Copilot July 15, 2026 21:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an opt-in, compile-time fault-injection layer (__FAULT_INJECTION__) to deliberately corrupt pointer/value reads at real profiler memory-access sites, so existing recovery paths (SafeAccess safefetch and walkVM setjmp/longjmp) are continuously exercised under test without affecting production builds.

Changes:

  • Added new fault-injection infrastructure (faultInjection.h/.cpp) with tiered probabilities and a per-thread PRNG seed/state.
  • Wrapped key memory-access sites (stack walker, HotSpot walkVM, VMStructs) with injection macros and added counters to observe injection/recovery activity.
  • Wired -PenableFaultInjection into native build configurations and gtest builds; added a dedicated gtest suite for enabled/disabled behavior.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
ddprof-lib/src/main/cpp/faultInjection.h New public fault-injection API/macros gated by __FAULT_INJECTION__.
ddprof-lib/src/main/cpp/faultInjection.cpp New implementation: PRNG, guard-region setup, poison address generation, injected int/long values.
ddprof-lib/src/main/cpp/threadLocalData.h Adds per-thread xorshift64 RNG state and accessors under __FAULT_INJECTION__.
ddprof-lib/src/main/cpp/stackWalker.cpp Wraps SafeAccess dereference sites with fault-injection macros.
ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp Wraps selected raw walkVM dereferences; increments longjmp recovery counter.
ddprof-lib/src/main/cpp/hotspot/vmStructs.h Injects faults in VMStructs::at() return pointers (rare tier).
ddprof-lib/src/main/cpp/safeAccess.cpp Adds counter increments on safefetch/safecopy recovery paths.
ddprof-lib/src/main/cpp/profiler.cpp Initializes fault-injection guard region during signal-handler setup (when enabled).
ddprof-lib/src/main/cpp/counters.h Adds new counters for safefetch/safecopy failures, injected faults, and walkVM longjmp recovery.
ddprof-lib/src/test/cpp/faultInjection_ut.cpp New unit tests for disabled identity behavior and enabled fault/recovery behavior.
build-logic/conventions/src/main/kotlin/com/datadoghq/native/config/ConfigurationPresets.kt Adds -D__FAULT_INJECTION__ to debug/release when -PenableFaultInjection is set.
build-logic/conventions/src/main/kotlin/com/datadoghq/native/gtest/GtestTaskBuilder.kt Mirrors the fault-injection define into gtest compilation under -PenableFaultInjection.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread ddprof-lib/src/main/cpp/safeAccess.cpp
Comment on lines 276 to 280
bool SafeAccess::handle_safefetch(int sig, void* context) {
ucontext_t* uc = (ucontext_t*)context;
uintptr_t pc = uc->current_pc;
if ((sig == SIGSEGV || sig == SIGBUS) && uc != nullptr) {
if (pc == (uintptr_t)safefetch32_impl) {
Comment thread ddprof-lib/src/main/cpp/faultInjection.cpp
Comment thread ddprof-lib/src/main/cpp/faultInjection.cpp
Comment thread ddprof-lib/src/main/cpp/faultInjection.h Outdated
Comment thread ddprof-lib/src/test/cpp/faultInjection_ut.cpp
zhengyu123 and others added 3 commits July 16, 2026 14:30
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit f1c6dc1)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124997579 Commit: f1c6dc1111fa56451484355bba2662323b3cd11b

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10321 ms (21 iters) ✅ 10408 ms (21 iters) ≈ +0.8% (±11%) — / —
akka-uct 25 ✅ 8949 ms (24 iters) ✅ 8862 ms (24 iters) ≈ -1% (±10.1%) — / —
finagle-chirper 21 ✅ 5903 ms (33 iters) ✅ 6000 ms (33 iters) ≈ +1.6% (±25%) ⚠️ W:3 / ⚠️ W:4
finagle-chirper 25 ✅ 5459 ms (36 iters) ✅ 5441 ms (36 iters) ≈ -0.3% (±24.4%) ⚠️ W:4 / ⚠️ W:3
fj-kmeans 21 ✅ 2733 ms (68 iters) ✅ 2788 ms (67 iters) ≈ +2% (±2.8%) — / —
fj-kmeans 25 ✅ 2821 ms (66 iters) ✅ 2800 ms (66 iters) ≈ -0.7% (±2.6%) — / —
future-genetic 21 ✅ 2046 ms (90 iters) ✅ 2065 ms (90 iters) ≈ +0.9% (±2.6%) — / —
future-genetic 25 ✅ 2064 ms (90 iters) ✅ 2059 ms (90 iters) ≈ -0.2% (±2.7%) — / —
naive-bayes 21 ✅ 1323 ms (130 iters) ✅ 1300 ms (132 iters) ≈ -1.7% (±32.1%) — / —
naive-bayes 25 ✅ 1013 ms (169 iters) ✅ 1005 ms (169 iters) ≈ -0.8% (±31.7%) — / —
reactors 21 ✅ 16153 ms (15 iters) ✅ 16163 ms (15 iters) ≈ +0.1% (±7.9%) — / —
reactors 25 ✅ 18820 ms (15 iters) ✅ 18126 ms (15 iters) ≈ -3.7% (±3.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 1 1981 / 2028 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 1 2179 / 2251 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 4 8560 / 8427 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 4 / ✅ 8460 / 8385 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ ✅ / 3 1270 / 1249 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 3 / 2 1284 / 1263 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 2904 / 2824 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 2 2744 / 2801 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 6 / 1 3543 / 3493 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 7 / 1 3479 / 3487 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 1698 / 1572 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 1960 / 1824 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit b88f34e)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125033402 Commit: b88f34ec8edeb4333ba152c49ebc16c24913e4d0

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 25): runtime +3.2% (2774→2864 ms)
  • 🔴 future-genetic (JDK 25): runtime +3% (2079→2142 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10294 ms (21 iters) ✅ 10207 ms (21 iters) ≈ -0.8% (±11.4%) — / —
akka-uct 25 ✅ 8806 ms (24 iters) ✅ 8847 ms (24 iters) ≈ +0.5% (±10.2%) — / —
finagle-chirper 21 ✅ 5993 ms (33 iters) ✅ 5964 ms (33 iters) ≈ -0.5% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5507 ms (36 iters) ✅ 5410 ms (36 iters) ≈ -1.8% (±23.7%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2746 ms (68 iters) ✅ 2751 ms (68 iters) ≈ +0.2% (±2.7%) — / —
fj-kmeans 25 ✅ 2774 ms (68 iters) ✅ 2864 ms (66 iters) 🔴 +3.2% — / —
future-genetic 21 ✅ 2097 ms (89 iters) ✅ 2095 ms (89 iters) ≈ -0.1% (±2.6%) — / —
future-genetic 25 ✅ 2079 ms (89 iters) ✅ 2142 ms (87 iters) 🔴 +3% — / —
naive-bayes 21 ✅ 1267 ms (135 iters) ✅ 1274 ms (134 iters) ≈ +0.6% (±32.7%) — / —
naive-bayes 25 ✅ 1012 ms (169 iters) ✅ 1007 ms (170 iters) ≈ -0.5% (±31.2%) — / —
reactors 21 ✅ 16315 ms (15 iters) ✅ 16117 ms (15 iters) ≈ -1.2% (±7.6%) — / —
reactors 25 ✅ 18032 ms (15 iters) ✅ 18381 ms (15 iters) ≈ +1.9% (±5.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 1 1961 / 1910 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 2 2182 / 2135 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 2 8807 / 8361 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 8192 / 8209 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 3 / ✅ 1255 / 1265 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 4 / 2 1301 / 1285 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 3 2941 / 2957 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 2 2834 / 2878 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 5 / 5 3488 / 3503 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / 2 1714 / 1536 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1876 / 1844 ✅ / ✅ ✅ / ✅

zhengyu123 and others added 3 commits July 16, 2026 18:32
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@datadog-datadog-prod-us1

Copy link
Copy Markdown
Contributor

Pipelines

Fix all issues with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

DataDog/java-profiler | get-versions   View in Datadog   GitLab

Code Quality Checks | codeql (cpp)   View in Datadog   GitHub Actions

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a9371e4 | Docs | Datadog PR Page | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants